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ABSTRACT 

Tests of reading comprehension presently used do not 
provide one important item of technical data: the extent to which 
questions used in the test could be answered without leading the 
paragraphs upon which those questions are based (paragraph 
dependency) . This leaves the test user guessing as to whether the 
students taking the test and performing well did or did not 
understand the written material contained in the test. Indices of 
paragraph dependency for five widely used standardized tests of 
reading comprehension were obtained. Five tests were administered to 
1200 Students each, not allowing these students to read passages. In 
addition, control data were obtained by administering the tests in 
their normal format to 600 students each. Students were selected from 
10 locations covering Indiana and i^ere equally divided over grades 4, 
5, and 6. The results indicated that none of these major tests 
provides sufficient guarantees against the answering of items cm the 
basis of information other than that presented in the passage. 
Average probabilities of correct responses with no passage present 
ranged between .32 and .50, well above the expected chance score of 
.25. (Author) 
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Preface 

Though I aci convinced that;, in the long run, our schools stand to 
benefit from the work done by so ijany educational researchers, it is the 
latter who, in most cases, benefit aost directly from any cooperation be- 
ta'/een the two. I am therefore deeply indebted to the personnel in the 
ten Indiana school systems tliat participated in this study and to the stu- 
dents in their schools. I vzish to thanlc all of them. In particu3ar, " im 
grateful to i(iy priiaary contacts in each of the school systems: Mrs. Helen 
McDaniel and Ilessrs. Leo Joint, Donald Eberly, David VThaley^ Donald ifessey, 
Herbert Reese, iielvin I<fozier, George Westfall, Charles Arvin and J. 0. 
Smith. 

Collecting as much data as we did is impossible without the underpaid 
help of marqr graduate students* I hope this enterprise v/as an education 
for at least some of them. Travelling across Indiana with big boxes of 
testing materials v/as most of the time hard work and only sometimes fun. 
I wish to thank my regular crev;: Carol Brooks, Dave DoTOing, Beverly Farr, 
P. J. Fitzgerald, Liary Halpin, Linda Hoyman and Dick Szuny. 

Finally, v/hat would I have done without the diligent v/ork of i^iary 
Ella Brady, Jeanne Burns and t5ary Halpin in the analysis, write up and run- 
off stage? Nothing! 
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INTEODUCl^ION 

The purpose of this study was to determine to what extent items in 
a number of selected standardized tests of reading can be answered with- 
out prior reading taking place. Indices of paragraph dependenqr were 
calculated* The study was liraited to so called tests of paragraph and, 
or story coiaprehension* 

Tests of reading coipprehension purport to measure how well a stu- 
dent understands what he is reading. Many of these tests employ ques- 
tions to ascertain the degree of this understanding. This technique is 
based on the tacit assumption that a direct relationship exists betvreen 
the reading of the passage or story and the answering of questions about 
it. In the case of a great many reading test items from standardized 
tests this is a faulty assumption. 

More than 25 years ago Davis (I9W1) asked the question: 'Vhat do 
reading tests really measure?'' His answer to his own question - an an- 
swer repeated in essence in his more recent study (Davis, I968) - indi- 
cated a definite dissatisfaction with the inability of standardized 
tests of reading to measure the skills "considered highly important by 
the authorities in the field (Davis, I9IA, p. 187)." 

The legitimacy of a concern for the functiciiing of reading tests as 
they are nm kno\m is underscored by a series of recent studies which re- 
vealed * -at in many cases successful performance on th* reading measure 
was only loosely related to the necessity for the reader to have read the 
passage on which the questions presumably v/ere based. 

A relatively detailed picture of students* ability to answer compre- 
hension questions without the aid of the text from which they are derived 



is provided by studies by Weaver and Bickley (1967), Biclvl^, Weaver and 
Ford (1968) and Weaver, Bickley and Ford (1969)* In a series of studies 
utilizing the black-out technique, one of the recurrent o^qperiaental con- 
ditions was that students were required to answer multiple-choice items 
sampled ftrom reading tests listed in the Si:rbh Mental Measurement Yea^^ 
book (Buros, I965), with the accompanying reading passages completely 
blacked out. In light of what one conventionally assumes about the func- 
tion of a reading test, their finding is somewhat startling: "The Ss who 
had no reading passage to aid in answering the items, nevertheless, cor- 
rectly completed 67?^ as many items as Ss with all the reading passage" 
(V7eaver and Bickley, 1967, p. 29!^). A further analysis of this phenom- 
enon led these authors to conclude: 

In other words, with the materials here, there is dif- 
ference betxreen having or not having a reading paragraph, even 
in the less relatedness of reading paragraph condition, but 
this effect is much more pronounced in the more relatedness to 
reading paragraph condition, (VJeaver, Biciaey and Ford, I969, 
p. 12) 

The above statement may be interpreted to mean that items of a ro'^v- 
tively more factual nature, to be answered directly on the basis of in-^ 
formation in the passage, are easier to ansi/er without the paragraph pre- 
sent than are items which are only indirectly related to the information 
in the paragraph - the inferential items. 

Weaver and Bickley (1967) suggest a number of possible ways in which 
the Ss could have answered the test items without aid of the relevant pas- 
cage: knov/ing the answer ftom prior learning; elimination of irrelevant 



distjractorsj the use of information embedded in preceding items. 
Samuels (I968) demonstrated that high associations among elements in 
an item stem and the correct distractor, too. facilitate ansxrering 
of reading comprehension items prior to reading the passage. 

In order to evaluate the consequences of the above findings^ it is 
necessary to recall the distinction betxreen a n^ ^essary condition and 
a sufficient condition (Carney and Scheer, 1S5U, p. 207). It can eas- 
ily be granted that reading involves relating whatever is being read * ^ 
prior experience. As such, prior learning is a necessary condition to 
reading, as long as a definition of reading includes a reference to un- 
derstanding. The statement that prior learning plays a legitimate role 
in the answering of multiple-choice questions subsequent to having read 
a passage is quite acceptable. Hoxrever, the fact that prior learning 
is a necessary condition for answering these items does not make it a 
sufficient condition. A reading test, for instance, is distinguishable 
ftom a listening test: in addition to the prior learning and knowledge 
present,- some reading* and not some listening, takes place before the, 
test is taken. In short, it seems reasonable to require that a reading 
test measure sets of behaviors which are functionally related to reading 
a passage. 

It must be clear that any measure of some variable operates best 
when irrelevant sources of information germane to performance on that 
measure are eliminated. Of the three sources of information listed by 
Weaver and Bicki.^ (196?) > none seems e::clusive to reading tests. H017- 
ever, whereas the elimination of the last two sources (irrelevant dis- 
tractors and related items) may require strategies common to test con- 



struction in general, the control of the fir fit source (prior learning) 
may be achieved in a way relatwelj^ unique to the area of reading test?. 

Reading, as a skill-centered area of instruction, is relatively 
content independent. That is, learning to read does not primarily mean 
to acquire a body of knovrledge but rather to master a set of skills. As 
a consequence, the maker of reading tests is relatively free of the obli- 
gation to have his tests represent information which embodies existing 
kna-aedge in a given area of human studies. In principle, there is no 
reason vzby reading skills cannot be tested with materials which actually 
represent modifications of commonly accepted statements of relationships 
between elements of reality. It is tliis freedom in the construction of 
reading tests which allows the test constructor to control the influeace 
of past learning to a greater extent than is possible in most othc.- .x*eas 
of testing for scholastic achievement. 

From the studies reviei^ed above, it becomes clear that, in few cases, 
test authors have been able to capitalize on this characteristic of read- 
ing measures. There is a great deal of evidence that the lack of pa;. ^ ;e 
control found by Weaver et al. is not limited to the test items which hap- 
pened to be selected into their instruments. 

Preston (iSSk) had 128 college freshmen take the first 30 comtprehen- 
sion items of the Cooperative English Test: Test C2, Reading Comprehen- 
sion (Higher Level), Form R, without the passages which the items were 
^Gupposecl to test. After taking the passageless test, Ss took the test 
in the conventional way. On the 30 items the e^qpected mean score was 6. 
The obtained score was 8.3*^ (p <.00l). A second interesting finding was 
that the ability to ans\/er questions Trithout passages had a lo;r corrLoXt-ticn 



with scores on the regular administration of the test (r = .20) and none 
at all with vocabulary (r = .IS, ms*)* 

Bloomer and Heitziaan (19^5) report findings which tend to substan- 
tiate this loay correlation found between answering questions with rele- 
vant information present and with that information absent. In their 
experiment, a groiip of eighth grade students took a pretest consisting 
of multiple-<ihoice questions, then read the reading passages and tooL 
the pretest as a post-test. a?he correlation between the scores was .12 
which was not significant (n=36). During the post-test, however, the 
information tos present only to the extent the student had memorized it. 

The study by Christ ensen and Stordahl (1955) is an exaii?)le of the 
difficulties in researching reading conqprehension that arise from the 
fact that the reading of the passages sometimes adds relatively little 
information. Their research attCT5>ted to deleimine the relative effec- 
tiveness of various organizational aids in comprehension and retention. 
In all, 35 treatments were administeredwithl2 8ULbjectsper treatment. 
Subjects were Air Force trainees. No significant differences were found 
among treatment group post-test means. The experiment, replicated TTith 
another reading passage, resulted in another set of nonsignificant dif- 
ferences. In their attemgpts to find an explanation for the results, the 
authors touched upon the possibility that something might have been 
wrong with their laaterials. Th^r did not, hoi/ever, compare pre- and 
post-test means. On the passage for which they reported detailed data 
the overall pretest mean was 88^^ of their post-test mean, indicating 
that hardly any information vyas gained by reading the passages. The 
tests, with a mean item difficulty near 505$, were of reasonable diffj . li 



Their research seems to have been aborted by the nature of the ques- 
tions. Those questions did not require reading as a necessary condi- 
tion. In this study of. comprehension , therefore, behaviors were stud- 
ied which were under relatively little control of the reading passages. 
The extent of this control* apparently, was a variable of unknam quanti- 
ty in this study, 

llhile the Weaver, et al. studies mentioned above were done with 
college students, iGtchell (I967) got comparable results i/ith fourth 
grade pupils using a different test (Gates Basic Reading Test). Note- 
worthy in Mitchell's study is that boys tri.th Im I.Q/s scored no wc-o 
on the '*passago-out" items than thqr did on a test which included the 
^passage. 

About kO years ago Eurich (1931) grappled with the issue basic to 
the present study. He constructed t\iO reading passages with 50 multiple- 
choice items each. Passage A was of a general nature, whereas passage B 
contained highly specific and exact material. The first observation of 
interest to the present discussion made by Eurich is that while for "after 
reading" the reliabilities of the two tests were in the same order of 
magnitude, thqr differed vastly for the 'before reading" condition, t'ith 
the coefficient for the B passage being very lov7. Seemingly, the nature 
of the content of the passages largely determined the results under the 
"no-passage" condition. No uniform conclusion regarding the function of 
test items under that condition seemed possible in Eurich *s case. (Here, 
as before, one must keep in mind that the "after reading" condition c^-^-^^j 
not imply actual presence of the passages while the items were being ax*- 
swered.) Further information of interest is the correlation betX'/een "be- 



fore" and "after" reading performance. For Eurich's test A this cor- 
relation equaled .37; for test B, Tlius, only between I3 and 20ffo of 
the variance of the scores before and after was accounted for by a com- 
mon factor. Unlike Christensen and Stordahl's (1955) study, Eurich's 
data revealed large mean differences bet\7een pre and post-tests. 

Tuinman (1970), as part of a study involving experimental items 
designed to be highly passage dependent, administered the first kO items 
of the Sequential Test of Educational Progress - Reading, Form 3A. T^o 
mean score obtained by 13k 7th, 8th and 9th graders was 20.C6 when the 
passages were presented and I3.66 when only the questions were given. 
Thus, the '"passage-out" score vas 3k% of the possible score and 68^ of 
the score under the '^ssage-in" condition. 

Farr and Smith (1970) administered 32 items from the Nelson-Denny 
comprehension test to college sophomores and students. Initially the 
items were administered without the paragraphs. After a 3-.week inter- 
val a retest followed with the paragraphs present. They found that for 
five of the items the number of correct responses under the '^passage-out" 
condition exceeded the number of right answers under the "passage-in" con- 
dition. Also, for 12 of the items the number of correct answers in the 
"passage-out" condition* eicceecled 50%. 

The studies reviewed above indicate that quite a few items on 
standardized tests have little passage dependency. The item that has 
a response probability in the passage-out condition of l/k, where k = 
the number of options, is rare indeed. Per force, the same holds true 
for the test whose mean score equals l/n, where n = the number of items 
when only the test items and not the passages are being administered . 
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Does this mean that therefore such items and such tests are invalid and 
of little use? Not necessarily. Lack of passage-dependency signals 
potential invalidity more than actual lack of vafidity. It must be 
clear that if indeed an item- is responded to without prior reading of 
the text or paragraph, that item constitutes an invalid measurement in 
the context of a reading comprehension test. However, from the fact 
that an item is answerable without such prior reading of the text does 
not folloi-/ automatically that Ss taking the test will indeed not read 
the text. For this reason low passage dependency is "merely" a threat 
to valid measurement and not proof of invalidity. 

In the light of the above comments it becomes of some importance 
to determine whether children indeed are tempted tz skip paragraphs 
when taking reading tests. Recently, an attempt was made to ascertain 
to what, extent children will engage in such potentially test invalidat- 
ing behavior as partial or complete passage skipping. 

In the first study (Tuinman, 1972a) , 60 si:cth graders were randomly 
assigned to one of four treatment groups having to read long passages 
(L) or short passages (S) paired with either passage dependent (D) or 
passage independent (l) questions. Thus, four treatment test booklets 
were constructed (LI, SI, LD and SD). The short passages were incorpo- 
rated in the long ones. The mean passage dependency of I questions was 
.58; that of the D items was .25. These statistics were obtained during 
a pilot study. 

The test booklets contained 20 cardboard pages. On the front of 
each page was a question, on the back of it the accon5)anying story. Ss 
were told to take the test in any fashion they wanted. The dependent 



variable of interest was the number of items ansr^ered v/ithout a single 
glance at the passages. Whereas the effect of passage length was not 
significant, the effect of item type was. The Ss skipped significantly 
more I-items than D-items. 

A second study (Tuinman, 1972b) employed the same stimulus 
materials in a slightly different experimental design. First, a time 
pressure variable was added. ("There is a time limit" vs. "There is no 
time limit"). Secondly, the potential effect of an artificial "set" 
due to long sequences oj? highly passage dependent items or highly inde- 
pendent items vras reduced by using a repeated measure design. To each 
subject a set of mixed I and D-items was administered. Again the I- 
items invited more passag;e-skipping. Though the mean "skip" score was 
lotir (2.5 out of a possible l6) the range of scores (0-10) indicated that 
individual students may woU invalidate their test and (in the case of 
I-items) get away vith it. 

From the above discu£^.sion it becomes quite clear that (l) indi- 
vidual students may produce responses which are not under control of the 
passage and (2) that standardized reading tests contain mapy items which 
reward rather than punish such behavior. 

In the past, test authors and publishers have given little attention 
to passage-dependency. Its desirability has been only sporadically- 
stressed by test reviewers. The intent of the current study therefore is 
threefold. 

Firsts attention is called to the degree of lack of passage dependea- ' 
cy by obtaining data on five major reading tests. 



Secondly, an attempt is made to produce reliable item validity 
statistics (in particular, passage dependency indices) by using sam- 
ples larger than those used in most of the research reviewed above. 

Thirdly, the shift in passage dependency of iteras and tests as t 
function of educational growth of the respondents is demonstrated by 
selecting Ss in three consecutive grade levels. 

PEOCEDURE 

Tests 

Tests V7ere selected for analysis in terms of passage dependency 
baaed on the following criteria : 

a. Comprehension should be measured by means of the passage- 
questions technique. 

b. Preferably one level of the test would be suitable for adminis- 
tration in grades h through 6. 

c. The length of the tests would allow students to finish within 
one hour. 

dc The test should be widely used on a national level. 

The final selection of tests used in the present study was as fol 

lows: 

Test 1 - Nelson Reading Test, Form A 

Number of items: 75 
Test 2 - California Achievement Tests, Level 3 • Form A 

Number of items: k2 
Test 3 - SRA - Achievement Series, Reading, Form E, Blue level 

Number of items: 60 
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Test k - Jfetropolitan Achievement Tests, Reading - Elementary Battery 
Form F. Number of items: k5. 

Test 5 - Metropolitan Achievement Tests, Reading - Intermediate Bat- 
tery, Form F. Number of items: 45 

Test 6 - lov/a Test of Basic Skills - Reading, Multilevel, Form 5. 
Number of items: 6o 
This list of tests requires some comments. First of all, it may be noted 
that tests k and 5 are actually only tvro different levels of the same 
test. This is a function of the fact that the Metropolitan did not meet 
criterion b: no one level of this test vas suitable for grades 4, 5 and 6. 
Therefore, it was decided to use the Elementaiy Battery v/ith the 4th 
grade and the Intermediate Battery with the 5th and 6th grades, a secc" ' 
comment which needs to be made regards tests 3 and 5. The multilevel SR/l con- 
tains far more items suitable for use in grades 4, 5 and 6 than can be 
answered within one hour. For this reason Test 3 constitutes a subset 
of SRA items. This subset was arrived at by random selection from the 
pool of suitable passages of as many passages as were needed to construct 
a reasonably long test. This procedure resulted in the inclusion of 60 
items in Test 3. A similar procedure was followed for Test 6. 

E:qperimefltal versions* of the tests were created by mimeographing the 
passages and the items separately. Thus, each test consisted of a pas -age- 
booklet and a question-booklet. Tn the question booklet references to the 
passage booiaet were made that indicated which passage should be read with 



The author wishes to thank Harcourt, Brace, Jovanovich, nic.. Science 
Research Associates, Inc.;; CTBAlcGraw Hill, Inc. and Houghton Mfflir. 
for permission to use their tests in this research. 
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Which items. The items were left intact, vith the exception of changes 
made necessary by the different print fonaat of the passages. For in- 
stance, instead of "The vord squash in line 22 means^" the item in the 
experimental form might read "The word squash in line 27 means." 

Subjects 

An attempt \ms made to secure a sample of kth^ 5th and 6th graders 
which was not atypical in any specific sense. For this reason cooperat- 
ing officials of the Indiana Organization of Elementary School Princi-'.-ils 
were asked to designate ten school systems (and a fe;; back-up systems) 
which together would be representative of the school ppptdation of the 
State of Indiana. The author recognizes that this procedure does not 
result in the kind of representativeness associated with random sampling. 
However, administrative and logistical barriers to doing research in 
school systems selected randomly fipom a pool of systems are so large as 
to result eventually in all kinds of concessions which tend to invali- 
date the original purity of the sampling plan. Secondly, the selection 
of Ss at this stage does not involve the creation of comparison groups 
which require random sampling for the purpose of guaranteeing the inter- 
nal validity of the research. Rather, to the degree that the actual sam- 
ple used in this study is atypical of any specific population, the results 
will merely lack in generalizability to that population. 

Figure 1 (page I3) contains a map of Indiana, and an indication cf 
which cities provided subjects for this study. Of the 10 school systems 
originally invited to participate, only one declined b-scause of involve- 
ment in another measurement oriented research project. Figure 2 shows a 
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Figure 1 
State of Indiana 
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S 



1. Valparaiso 

2. LaPorte 

3. Warsaw 
h. Elkhart 
5* Madison 



6 . Columbus 

7. Indianapolis 

8 . Lebanon 

9 . Crawf ordsville 
10* Shell^inrille 



listing of the systems in the final saniple, the number of schools in each 
system and the total nuniber of students . 



Figure 2 

School Systems, Number of Schools, and Students Per System 

Schocas Students 

1. Valparaiso Community_Schpols 7 928 
Valparaiso, Indiana 

2. LaPorte Community School Cor- I3 1716 

poration 
LaPorte, Indiana 

3* Warsaw Community School Cor- 9 1227 

poration 
Warsaw, Indiana 

h. Elkhart Community School Cor- If 822 

poration 
Elkhart, Indiana 

l^dison Consolidated Schools 7 101^5 

Madison, Indiana 

6. Bartholomew Consolidated School 7 133I1 

Corporation 
Columbus, Indiana 

7. Metropolitan School District of 3 720 

Periy To;«iship 
Indianapolis, Indiana 



867 



8. Lebanon Community School Corpora- k 

tion ^ 
Indianapolis, Indiana 

9. Crawfordsville Community School 15 ztqo 

Corporation 
Crawfordsville, Indiana 

10. Shelby Eastern Schools 2 00 

Shelby ville, Indiana ^ 



The original design of the study called for 300 students per grade 
per school system. This quota could not be met by all systems in- 
volved. In addition, the larger systems welcomed testing of as many stu- 
dents as were available in the cooperating schools rather than leaving 
some classrooms out. The resulting distribution of subjects across the 
various systems , in fact, offsets the apparent overrepresentation of 
systems in rural communities someifhat, since the fe^7 school systems con- 
tributing the most subjects are situated in bhe more industrial northern 
region of Indiana, 

Table 1 gives the number of students per school system, per grade, 
per test and per test condition. Additional comments on this table vill 
be provided in the nesrt section of this report. 

It may need mentioning that all students present on the day of test- 
ing in a particular school or classroom were included in the study. The 
only e:cception to this is some 25 children (in a sample of over 9,000) 
who did not participate because the teacher advised against it on the 
basis of over-anxiety or extreme inability to read. 

Procedure 

The administration of the tests took place during the latter half of 
February, Iferch and the first half of April, 1972. 

A team consisting of the author and three to five graduate research 
assistants administered the tests. To insure uniformity of test admin- 
istration, the assistants were all trained in the procedure follor/ed i • 
administering the tests in order to standardize the procedures as much as 
possible. 
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Since the picrpose of the study vas to obtain passage dependency in- 
dices on all items in the tests used, no tiice limits were enforced. The 
standard directions used for the purpose of this study included: 

a. Mentioning of the fact that the tests vrere administered for the 
purpose of getting information on the tests and not on the chil- 
dren. 

b. The statement that the results would not appear on grade cards, 
or be reported to the teachers. 

c. A plea for cooperation. 

d. An explanation of how to use the test booklet with the passage 
booklets. 

e. The encouragement that "many questions can be answered without 
reading the stories." (Only for the children under the No-passage 
Condition). 

f. The announcement that there would be plenty of time. 
Depending on the test, the administration of the tests under the 

Passage condition lasted typically ft:om li5-6o minutes. Under the Ko- 
Passage conditior^ about 20-^5 minutes vrere needed. 

Cooperating schools were given the option to have their students 
tested in large groups in cafeterias, ebc. or in classrooms. This deci- 
slon was made on the basis of therestLLts of Ingle and DeAmico (I969) who 
found no effect of plysical conditions on standardized achievement test 
scores. The conditions contrasted in their study were "relatively poor 
physical conditions in an auditorium" and "relatively adequate physical 
conditions in regular cleasrccms." The principals of the schools in the 
present study, in general, preferred testing in classrooms. Thus, only ap- 



proximately ten percent of all test administrations took place in an 
auditorium or cafeteria. 

As indicated, two thirds of Ss took the tests without the passages. 
Assignment to the Passage condition (P) or ITon-passage condition (iSP) was 
done"with the classroom as the unit. Tlie argument for this decision was 
that the confusion resulting from the differences in time needed for com- 
pletion of the task and the necessity for tt/o sets of directions if both 
K and WP students would be present in one classroom would outi/eigh any 
advanti.\ges due to using the student as the unit of assignment. 

Responses were recorded on machine scoreable ansv/er sheets. Great 
care was taken to insure that students knew hot/ to use these. Infrequent 
problems in this respect were detected early, since, routinely, both the 
classroom teacher and the E monitored during the first ten minutes of the 
test administration. 

EESULTS 

1. Results Cocobined Over Grades 

Table 2 (See page 19) summarizes the scores on all tests across the 
three grades. This table invites a few comments. First of all^ it is 
clear that deviation ffom the publishers' standard test administration 
procedure had little effect on the reliability of the measurements. 
With the exception of Test 6 (a subset of items of the ITBr» all reliabil- 
ity coefficients under the P-condition are equal to or above .90. The 
fact that under the ISP-condition the KR-20*s are lower is not surprising. 
After all, in this condition the task is to guess at the answer. What is 
surprising is that the reliabilities remain as high as they do. This in 



Table 2 



Means, Standard Deviations, 101-20 Coefficients and Standard 
Errors of Measurement for the P and NP 
Conditions Across Grades 



Condition 


Test 


k 


X 


S.D. 


ICR-20 


SE 

m 




1 


75 


1^5.96 


15.8 


.96 


3.3 




2 


lt2 


26.66 


8.1 


.90 


2.6 


Passage 


3 


60 


37.17 


12.3 


.93 


3.2 


k 


h5 


29. 51^ 


9.1^ 


.92 


2.7 




5 


h5 


28.82 


8.7 


.90 


2.7 




6 


1|2 


27.03 


7.7 


.88 


2.7 




1 


75 


29.36 


6.7 


.67 


3.9 




2 


1(2 


1U.36 


k.l 


.51 


2.9 


Hon-Passage 


3 


60 


22.17 


S.3 


.70 


S.5 


k 


h5 


22.27 


6.7 


.81 


3.0 




5 


h5 


20.27 


5.0 


.65 


3.0 




6 


k2 


19.29 


5.0 


.68 


2.9 



itself is an indication that the behavior measured is not a random selech^ 
log of any of four multiple -choice options. 

The mean scores under the P-condition are in the expected range, 
typically some Gofo of the highest possible score. The decision to al- 
low more time than the test manuals specify, hotrever, makes it impossi- 
ble to interpret the scores of the P-students in terms of the norms pro- 
vided in the manuals. 

From Table 2 it can already be seen that none of the tests produces 
mean scores under the KP-condition close to T/hat one would escpect on the 
basis of chance only. For all tests, with the exception of Test 2, this 
chance score equals r/U, where r s the number of items. Test 2 contains 



20. 



a few five choice items j the chance score for this test equals 10.10. 
Table 3 details the esctent to which the scores under the passage con- 
Table 3 

lieans Under the JlP-condition Expressed as (l) Percentages 
of the Number of Items in the Test and (2) As 
Percentages of the Means Obtained Under 
the P Condition 



Test 


V 




%p as % of 
Kuniber of Items 


Chance 
Score {%) 


Xfjp as % 


1 


45.96 


29.36 


39 


25 


6k 


2 


26.66 


14.36 


3^ 


2h 


5h 


3 


37.17 


22.17 


37 


25 


60 


h 


29.51^ 


22.27 


50 


25 


75 


5 


28.82 


20.27 


i^5 


25 


70 


6 


27.03 


19.29 


1(6 


25 


71 



dition exceeded chance scores. The entries in the cells can be con- 
trasted directly with those in column 5? representing chance scores. 
It is clear that none of the tests even approxiraates the chance score 
under the KP-condition. Teets 5 and 6, inparticular, shew a high de- 
gree of passage independency. The fourth graders to whom Test h was ad- 
ministered managed to answer correctly 50^ of the items even though they 
never read the passage upon which the items were based. Tests 5 and 6 
fare little better and even Tests 1, 2 and 3 result in "guessing" scores 
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which are far above the level of statistical chance. 

The entries in the last column are even more startling. Of the six 
tests, three allow a student who does not have the passages to obtain a 
score as high as JOfo of what a student with the passages would get. On 
the average, for these tests, not reading the passage results in a loss 
of performance less than SOfo, Tests 1, 2 and 3 present only a slightly 
more reassuring picture, it may be noted that if one takes 60% of the 
riuniber of items as a typical mean score for multiple choice tests, the 
expected chance score of 2% represents approximately kOfo of the score 
obtained und.er the P-condition (60?5) . Again, Test 2 shows up more fa- 
vorably than the other tests. 

2. Results by Grade 

Tables k through 13 (See pages 22-26) contain the results presented 
above broken down by grade. Passage dependency of items is not a static 
characteristic. It varies with the test user. This is a potential prob- 
lem ir one particular test form is used for a number of grade levels. 
The nature and the extent of the problem are illustrated below. 

Table k needs little commentary except to note that there is an in- 
crease of the means across grades in both the P and the KP-condition. 
Whereas this is not surprising in the former case, the increases under 
the rip-condition are of some interest. The data indicate that a particu- 
lar item may be sufficiently passage' dependent at the lower level of the 
grade range for which the test was intended but insufficiently passage de- 
pendent at the higher levels. 
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Table k 

Test 1 - Means, Standard Deviations, KR-20 Reliability Coefficients 
and Standard Errors of Measurement for P and NP-Conditions. 
(Number of Items = 75; Chance Score = 18.75.) 
Results by Grade. 



Condition 


Grade 


X 


S.D. 


KR-20 






1* 


39.69 


16.5 


.96 


3.3 


P 


5 


47.58 


14.0 


.95 


3.2 




6 


53.1j6 


13.0 


.94 


3.1 




1* 


28.7lf 


e.k 


.64 


3.9 


KP 


5 


28.86 


6.9 


.69 


3.8 




6 


30.53 


6.8 


.67 


3.9 



Table 5 

Test 1 - Means Under the KP-Condition Expressed (1) as Percentages 
of the Kumber of Items in the Test and (2) as Percentages 
of the Mean IMder the P-Condition. Results by Grade. 



Grade iieans as % of Total iMean as % of Means Under 

KuEiber of Items Passage Condition 



4 38 ' 72 

5 38 61 

6 41 57 



Table 6 



Test 2 - Means, Standard Deviations, i(R-20 Reliability Coefficients 
and Standard Errors of Measurement for P and MP-Conditions. 
(Number of Items = U2; Chance Score = 10.10) 
Results by Grade. 



Condition 


Grade 


X 


S.D. 


KR-20 


SE 
111 




k 


23.05 


7.18 


.85 


2.77 


P 


5 


28.04 


7. hi 


.88 


2.55 




6 


28.92 


8.25 


.91 


2.47 




h 


13. 3*^ 


3.83 




2.88 


DP 


5 


lh.33 


4.10 


.51 


2.87 




6 


15. U2 


k.2k 




2.88 



Table 7 

Test 2 - Means Under the NP-Condition Escpressed (1) as Percentages 
of the Number of Items in the Test and (2) as Percentages 
of the Mean Under the P-Condition. Results by Grade. 



Grade Ueeins as "fo of Total ifean as i of Means Under 

Number of Items Passage Condition 

^ 32 53 

5 3h 51 

6 37 53 
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Table 8 

Test 3 - Means, Standard Deviations, ICR-20 Reliability Coefficients 
and Standard Errors of Measurement for P and NP-Conditions. 
Niuaber of Items = 60; Chance Score = 15.OO) 
Results by Grade. 



Condition 


Grade 


X 


S.D. 


KR-20 


m 




h 


31.72 


U.49 


.92 


3.29 


P 


5 


38.76 


12.03 


.93 


3.14 




6 


41.25 


11.51 


.93 


3.05 




h 


19.97 


5.29 


.58 


3.4l 


NP 


5 


21.91 


6.06 


.67 


3.45 




6 


24.61 


5.50 


.72 


3.46 



Table 9 

Test 3 - Means Under the NP-Condition Expressed (l) as Percentages 
of the Number of Items in the Test and (2) as Percentages 
of the Mean Under the P-Condition. Results by Grade. 



Grade Means as ^ of Total I^n as % of Means Ibider 

Kuniber of Items Passage Condition 



h 33 ^ 63 

5 37 57 

6 kl 60 
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Table 10 

Test 5 - ileans, Standard Deviations, 10^-20 Reliability Coefficients 
and Standard Errors of Ifeasurement for P and KP-Conditions . 
(KuBiber of Items = k^i Chance Score = 11.25) 
Results by Grade. 



Condition 


Grade 


X 


S.D. 


KR-20 


SE 

m 




k 


29.5^^ 


9.kl 


.92 


2.67 


P 


5 


27.19 


8.77 


.90 


2.80 




6 


30.43 


8.32 


.90 


2.65 




k 


22.27 


6.70 


.81 


2.96 


IJP 


5 


19.16 


k.Ql 


.62 


2.99 




6 


21.45 


4.86 


.63 


2.95 



4 



Table 11 

Tests k and 5 • Means Under the KP-Condition Expressed (1) as Percentages 
of the Nuinber of Items in the Test and (2) as Percentages of the Mean 
Under the P-Condition, Results by Grade, 





Grade 


Test 


Means as 'j!} of Total 


ilean as % of Ileans Under 






Number of Items 


Passage Condition 


h 


h 


50 ' 


75 


5 


5 


43 


70 


6 


5 


48 


70 
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Table 12 

Test 6 - Means, Standard Deviations, KR-20 Reliability Coefficients 
and Standard Errors of Measurement for P and NP-Conditions . 
(Number of Items « k2i Chance Score = 10.50) 
Results by Grade. 



Condition 


Grade 


X 


S.D. 


KR-20 






h 


23.75 


7.92 


.88 


2.7h 


P 


5 


27.82 


7.55 


.88 


2.61 




6 


29. k7 


6.56 


.85 


2.58 




h 


17.85 


4.82 


.65 


2.85 


HP 


5 


19.3*^ 


t^.95 


.67 


2.85 




6 


20.67 


4.96 


.68 


2.81 



Table 13 

Test 6 - Ileans Under the KP-Condition E3q>ressed (1) as Percentages 
of the Number of Items in the Test and (2) as Percentages 
of the Mean Under the P-Condition. Results by Grade. 



Grade 



Mean as ^ of Total 
Nuniber of Items 



Mean as ^ of Means Under 
Passage Condition 



6 



h2 
h9 



75 
70 
70 
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The entries in the second column of Table 5 indicate the absolute increase 
in passage independency of the test items as the student becomes more 
sophisticated. The entries in the third column are of particular inter- 
est. They index the performance under the HP-condition relative to that 
under the P-condition for a particular grade group. The decrease in the 
percentages indicates that the means under the P-condition increase fast- 
er than those under the NP-condition. In summy, the data in Table 5 
may be interpreted as follovis. Whereas a student in the 6th grade who 
fails to read the passages (or some of them) can get more answers right 
than a Ifth grader in the same position, the score of the 6th grader rela- 
tive to those of his peers id.ll be lower than the score of the I|th grade 
student when compared to the scores of other kth graders. 

Tfeibles 6-13 shm patterns similar to the ones discussed for Test 1 
for the remaining tests. It may be pointed out, hoT7ever,that the in- 
crease in relative passage dependency doe^ in general* not hold for grades 
5 and 6. This can be seen from inspection of the entries in the last 
column of Tables 7, 9> U and 13. Tito of the tests, 2 and 3, even ahov a 
reversal; it is too small to be of any iuiportance, however (Tables 7 and 
9)* Care must be taken not to misinterpret the data in Tables 10 and 11, 
where the combined results for Tests k and 5 are reported. Of special 
interest is the very high reliability coefficient for test k under the 
NP-condition. Table 11 reveals that Ss in this condition obtained a 
mean score as Mgh at 73% of the mean score tinder the P-condition. The 
items on this particular test allowed the IIP sub;)ects to employ a highly 
reliable and effective response strategy. 
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The entries in the column headed "Means as ^fo of the Number of Items" 
of Tables 7, 9 and 13 z\m the same absolute increase in passage indepen- 
dency as a function of increased sophistication of the respondents^ as was 
noted for Test 1. For Test % the comparison by grade is limited to grades 
5 and 6 (Table !!)• 

Items With Higher than Chance Scores 

There are two vmys in which the mean score of a group of subjects 
under the KP-condition can be higher than r/k, when r = the number of 
items and k = the average number of options per item. First> there ma^ 
be a relatively small group of very easy items. Secondly, there may be 
a large group of moderately easy items, all of which, ha/ever, have a 
probability of being passed larger than l/k. 

Additional light on the mean scores reported in Tables I1-13 is sup- 
plied by the information in Table ik (See page 29) and Table 15 (See 
page 30) • The 1^ and 5?5 upper confidence limits (one-sided) were cal- 
culated for each group of respondents to each test. The basis for the 
calculations is the binomial distribution where p = i/k and C.L. = l«p. 
The limits were computed around the quantity l/k, in most cases equal to 
.25* An item is said to have a passage independency larger than l/k, if 
the observed item difficulty exceeded the upper confidence limit. Table 
Ik shov7s, for each test, the number of items per test vdth a passage in- 
dependency larger than l/k. In Table 15 the same information is expressed 
as percentages of the number of items per test. 

Table 15 in paxticxilar points up a few interesting characteristics 
of the tests analyzed • First of all, it becomes clear that, in general 
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Table 11^ 

Number of Items Per Test with a Difficulty Under the KP-Condition 
Higher than l/k, VJhere k = the Number of Options Per Item. 
Items Included Had an Observed Difficulty Exceeding 
One-sided Upper Confidence Limits Around (l/lc) . 
The liain Entries are Based on 5'^^ c.L.'s; 
the Entries in Parentheses are Based 
on 1% C.L.'s. 



Test # of Items Grade h Grade 5 Grade 6 

1 75 h9 kQ 51 

m m ik9) 

2 k2 25 26 25 

(22) (25) (210 

3 60 28 32 ho 

(25) (29) (37) 

k k5 37 _ 

(37) (~) (-) 

5 U5 — 32 33 

(-) (30) (32) 

6 k2 31 3U 33 

(30) (33) (33) 



Table 15 



Percentage of Items Per Test ^rt.th a Difficulty Under the MP-Condition 
Higher than l/k, T-Jhere k = the Nuniber of Options Per Item. Items 
Included Had an Observed Difficulty Exceeding One-sided Upper 
Confidence Limits Around (l/k). The Ilain Entries are 
Bp-sed on 5^ C.L.*s; the Entries in parenthesis 
are Based on 1^ C.L.'s. 



Test Grade k Grade 5 Grade 6 



1 


65 


6k 


68 




(61) 


(60) 


(65) 


2 


60 


62 


60 




(52) 


(60) 


(57) 


3 


hi 


53 


67 








(62) 


k 


82 








(82) 




(-) 


5 


(~) 


71 


73 




(67) 


(71 


6 


69 


76 


73 




(67) 


(73) 


(73) 



tests with the highest mean scores (relative to the number of items) under 
the HP-condition also have the highest percentage of items iTith a passage 
independency index larger than l/k* This can be seen by contrasting 
arable 15 ^rith the entries in Tables 5, 7, 9y 11 and 13* Test tor in- 
stance, had an HP mean score which \ms 50^ of the total number of items in 
this test (Table 11). From Table I5 it can be seen that of those 
items liad a NP-diff iculty index exceeding the Vjo iqpper confidence limits 
around l/k,\7here k = the average number of options per item* A compari- 
son of the results for Tests 2 and 3j hot^ever, shovrs an exception to this 
general finding. From Tables 7 and 9 it can be seen that Test 2 has W 
mean scores which are, if e:cpressed as a proportion of the total niamber 
of items in the test, smaller than the NP mean scores for Test 3. Yet, 
for grades h and 5 the percentage of items with a passage ind^endency in- 
dex larger than l/k is higher for Test 2 than it is for Test 3. Test 3, 
relative to Test 2, is a test where a high UP mean score is obtained with 
relatively few passage independent items. Tables Al - A6 contain the 
HP-difficulties for each item and a designation in regard to whether or 
not these exceeded y}^ and 1^ C.L.'s around l/k, where k is the number 01 
options per item. 

Passage Dependency Indices 

The data presented above niakes clear that generally quite a ferr 
items allow respondents to answer correctly wl.en they have not read the 
material iq)on which the items purportedly vrere based. Tlie degree to which 
an item requires reading of the passage has been referred to as that item*s 
passage dependency. The term passage independency has been used to indi- 



cate the .(relative) lack of passage dependenor. ^Jhen the question arises ^is to 
how to index numerics IX;'- passage dependency of an iteo, the most logical 
cotirse of action seems to be to obtain an estimate of the proportion of 
respondents that can answer the item under the KP-condition. Thusj 

Passage Dependency Index 1 = The proportion of correct 

responses under the NP con- 
dition 

Thus, the Imer FDl^ is, the more passage dependent the item for which 
it was calculated. Since the concept of 'the Imer the better" may be 
slightly conftising, it is better to calculate: 

PDI = 1- - EDI, 
2 1 

This inde:c v/iU increase as passage dependency increases. 

Theoretically, PDI^ can take any value between l/k and 1.00, where 
k = the number of options per item. For most multiple^choice tests the 
range of PDI^ vrould be .25 - 1.00. Hov/ever, certain items have charac- 
teristics which lo^xer the probability of choosing the correct response 
under the NP-condition. Actual values of PDI^^ loizer than .25 may there- 
fore be observed. Conversely, PDI^ values higher than the theoretical 
.75 do occur fteq[uently. 

Table 16 (See page 33) contains the PDIg values for the tests. 

In Tables 31 ^ 32 (See Appendix B) the PDI^ (d^^) and PDIg values 
for all the items have been listed. 

There is a problem in interpreting PDI^ and PDIgj hoxrever. Consider 
the following statistics for t^ro items. Item 1 has a difficulty d^^ under 
the P condition of .35 and a difficuUy djjp under the condition of 
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Table 16 

H)l2 Values for Tests 1-6; 
By Grade 



Test 




k 


5 


Grade 

6 


7 


1 


62 


62 


59 


61 


2 


68 


66 


63 


66 


3 


67 


63 


59 


63 


h 










5 




57 


52 


55 


6 


58 




51 


5h 



.35 also. Item 2 has a d^ equal to .75 and, like item 1> ^ d^^ 
value of .35. For both items PDI^ = .65. However, while it seems dif- 
ficult to use item 1 for observing any behavior controlled by the pas- 
sage, item 2 can be used to this end. After all, at least k<yjl> of cor- 
rect responses found their sources in the passage. (The question of 
correctly guessing is left aside for the moment.) For this reason, 

✓ 

while PDI^ pro\'ides some information about an item's passage dependency, 
it does not tell the whole story. Tuinman (I97O) proposed the ratio of 
V'^S ^ 'oetter index of the degree to which a question can be used 
efficiently to measure responses based on reading the passage. Tlius, 



For item 1 in the exaaiple above E = 1.00 and for item 2 Ei = .50. 

1 1 • ^ 

For convenience of interpretation E^ is proposed : 

2 IIP' p 

Table 17 contains the average 6^ and dp values for the six tests and 

the resulting E values. 
2 

Table 17 

Average Difficulties Under P and KP Conditions and Values 
for Six Tests; Combined Across Grades 



Test 


% 




^2 


1 


.61 


.39 


.36 


2 


.63 


.31* 


.1*6 


3 


.62 


.37 


.1*0 


1* 


.66 


.50 


.21* 


5 


.6k 


M 


.30 


6 


.6k 


.1*6 


.28 



Since the six tests have conrparable difficulties under the P-condition, 

Eg is of importance in particular for comparison of items of tests with 

different d values. Tables Bl - b6 contain the d^, d , PDI and E 
^ ^ HP 2 2 

values for all the items. 



35. 

The indices proposed above are all very siraple and make no special 
assumptions. In Appendijc c a number of other indices of passage depen- 
dency are discussed. Tables ci - c6 contain the values of these statis- 
tics for all items. 

Discussion 

This report has a two fold purpose. First, it intends to high- 
light the problem of passage dependency of reading comprehension items; 
secondly , it is meant to be a working document for those who desire to 
do further analyses on the items included in the tests used. For this 
purpose extensive data tables liave been included in Appendices A and B. 

From the data presented above, a number of major conclusions can 
be drawn. First of all, it appears that commercially marketed tests of 
reading comprehension vary considerably in the degree to wMch their 
items are passage dependent. This points up the need to consider pas- 
sage dependency when choosing among various tests. Everything else be- 
ing equal, the test with the most items with tne highest degree of pas- 
sage dependency offers the largest guarantee against invalidity due to 
responding to items without prior reading of the passage on which the 
item is based, a caution is in place, hmever; passage dependency mpy 
be purchased at a price that the test consumer is unwilling to pay. 
The Weaver, Bickley and Ford,(l969) study, for instance, indicates that, 
generally, inference items are moi^e passage dependent than factual items. 
In addition to considering passage dependency, the consumer must satisfy 
himself in regard to the content validity of the test under consideration. 
Secondly, it becomes obvious ft-om the data presented above, that none of 
the five tests approaches passage dependency close to optimal limits. 
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This is not so surprising in view of the ftict that it is extremeOy dif- 
ficult to construct highly passage dependent items, even if the passages 
contain highly imaginary materials (Tuinman, 1970). Thirdly, as expected, 
(Tuinman, 1971), the degree to which items are passage dependent is a 
function of the age, c. q. educational sophistication of the child that 
takes the test. Tests in this study, designed for grades k-S, showed a 
consistent decrease in passage dependency from fourth graders to sixth 
graders. This fact must be kept in mind hy the test user who decides to 
select a test with a wider grade range. 
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APPENDIX A 



Tables (A1-A6) Per Test for Item Difficulties 
Under the NP-Conditions , Per Grade. 



Table Al 



Test 1 - Item Difficulties Under the NP-ConAitions, Per Grade. No Aster 
isk Indicates that the Item Difficulty Exceeded the 1% Upper Conf iden -e 
Limit Around (lA). One Asterisk Indicates that the Difficulty was 
Higher than the 5% C.L. but not Higher than the 1% C.L. Two Aster- 
isks Indicate that the Difficulty did not Exceed the 5^ C.L. 
(Decimal Points have been Deleted.) 



Grade Grade Grade Item Grade Grade frade 
Item 456 456 

lo 

41 
42 

4^ 

^5 
46 

47 
48 
49 
50 
51 
52 

5 
55 
56 
57 
58 

59 

60 
61 
62 

el 

65 

66 

67 
68 

69 
70 
71 
72 
73 
7^ 
75 



1 


82 


80 


83 


2 


?1 


?0* 


?1 




49 


•50 






89 


82 




5 


^0 


^6 




6 


77 




90 

(y 


7 


?7 






8 


71 


66 




9 




46 




10 




18** 


18** 


11 


66 


66 




12 


60 


63 


68 


13 


69 


67 


66 


14 


78 


81 


84 


15 


51 


56 


61 


16 


60 






17 


66 


59 


60 


18 


09** 


06** 


08** 


19 


45 


42 




20 


32 


30 




21 


54 


51 


54 


22 


39 


3J 




23 


07** 


06** 


06** 


24 


50 


55 


59 


25 


12** 


13** 


iZf^:* 


26 


28** 


24** 


28** 


27 


29* 


25** 


34 


28 


50 


52 


56 


29 


19** 


21** 


21** 




45 


42 


40 ' 


31 


58 






32 


48 


11 






56 


56 


59 


3^2 


29* 


41 


50 


5| 


44 


50 


54 


36 


22** 


22** 


24** 


37 


28** 




31 


38 


^5 




52 



39 


40 


42 


19** 


19** 


17** 


?Q** 

✓ 


32 






^ f 


36 


26** 


o-a** 


29"* 
fc» 7 




16** 




IS** 


Ik** 


lllK-* 


31 




39 




26** 


31 


'?6 


•^7 


37 


^7 


^7 


24** 


18** 




50 


50 


^2 


39 
44 


^2 

2^4** 


5i 
46 


28** 


24** 


23** 


33 


34 






66 


67 


60 


64** 


67 


19** 


16* 


20** 


21** 


29** 


29* 


28** 


26** 


24** 


20** 


21** 


20** 


46 


48 




41 


38 




21** 


22''^* 


2^*" 


13** 


20** 


19*'^ 


19** 


16** 


^-3** 


22** 


20** 


27*vc 




40 


4-) 


^9 


47 


5i 


15** 


12«* 


11** 


19** 


17** 


14 ;t* 


42 


48 


52 


29* 


27** 


3^* 


22** 


22** 


22** 


32 


28** 


28 >* 



Ill 



Table A2 

?v,;.^*r ^iJ^i'^S*^?^ "^'^^^ NP-Conditions, Per Grade. No Ast- 
lLJ? I M Difficulty Exceeded the 1% Upper Confiderxc 

iy^h Asterisk Indicates that the Difficulty was 

Higher than the 5% C.L. but not Higher than the 1% C.L. Two Aste?- 
. isks Indicate that the Difficulty did not Exceed the 5% C,L, 

(Decimal Points have been Deleted.) 



ite. °r ite. °r °T 



\ IP It:: u:: - ij. i« 

5 23^* 22** 17** 26 26*^'^ Je** iJ-:* 

? r r 3^'* II p. u I 

51 

ti I 1% tt I i if I 



" i,^ 

11 ^/i Ve. ?} 31 29* 24** 2^ 

" 32 2' 5? 32 30' ^1 

i i ^ ' ^ 6? 

itf ?° ^8 35 2^** -^1 00 

II I g:: " 



27** 



18 -^Q Zlo -^^ -^^ 16** 17 

" ^2 ^6 p ^ i9« 

20 23»» ?j 2? 21 21** 

3i t % 23- 20^ 23« 



Table A3 

Test 3 - Item Difficulties Under the NP-Conditlons, Per Grade. No Aster- 
isk Indicates that the Item Difficulty Exceeded the 1% Upper Confidence 
Limit Around (lA). One Asterisk Indicates that the Difficulty was 
Higher than the 5% C.L. but not Higher than the 1% C.L. Two Aster- 
isks Indicate that the Difficulty did not Exceed the 5% C.L. 
(Decimal Points have been Deleted. ) 



Grade Grade Grade Grade Grade Grade 



1 


57 


2 


80 




61 


I 


6k 


5 


11** 


6 


50 


7 ' 


48 


8 


37 


9 


17** 


10 


i^8 


11 


2^** 


12 


34 


i2 


25** 




72 


15 


19** 


16 


37 


17 


54 


18 


32 


19 


23** 


20 


26** 


21 


42 


22 


46 




66 


1^ 


5^ 


25 


20** 


26 


28** 


27 


47 


28 


14** 


29 


^5 


30 


19** 



61 67 31 15** 19*» 15 

80 85 32 23** 29* 35 

56 60 33 26** 29* 34 

59 69 3^ ^7 59 67 

14** 18** 35 i7«* i8*« 22** 

48 56 36 22** 33 39 

56 60 37 25** 23** P4** 

41 52 38 22** 26** '8** 

23** 28** 25** 36 V 

^7 50 4o 30* 33 4? 

25** 20** 41 24** 22** 21** 

^1 ^8 42 31 37 

28** 29* 43 30* 30* 31 

75 80 44 29* 28** 28** 

30* 40 45 21** 22** 23** 

5^ 59 46 51 57 67 

61 62 47 26** 28** 29*^ 

27** 36 48 20** 21** 2r " 



29* 35 49 27** 23** 25** 

28** 28** 50 28** 28** 34 

47 51 38 40 51 

48 55 52 19** 21** 26*- 

11 79 53 3^ 39 41 

65 82 54 21** 24** 27** 

20** 23** 55 22** 27** 30* 

26** 30* 56 29* 26** 37 

59 7^ 57 23** 25** 24** 

17** 20** 58 23** 25** 2 7* 

53 58 59 26** 28** 30 

25** 29* 60 25** 19** 23 



* 
* 



Table A^f 



f^J ; r.^^f" Difficulties Under the NP-Condltlons, Per Grade. No Aster 
^^JS^i^^'^^J ^':^yt^''^^ri''^'' Difficulty Exceeded the 1% Uw'; Con?ldlJce 
mit lyH'^ Asterisk Indicates that the Dlf ficulty Jas 

^!^®^*^f" l^J^ "° ^^S^^^ ^^an the U C.L. Two AstS! 

isks Indicate that the Difficulty did not Exceed the 5% C.L. 
(Decimal Points have been Deleted.) 



Grade 

■ ^ Item 



1 51 23 

? 79 25 70 

: 81 il ^6 

I ^? 27 46 

° ^1 28 zl< 



8 81 



Grade 
4 



18** 
72 



16** 



3? 57 



10 74 ?2 al 

^2 T A 

37 ^? 

^7 2i 38 68 

21 4l 43 ^7 

22 39 ^1 

^5 38 



hk 



Table A5 



r Dirricultles Under the NP-Conditions, Per Grade. No .'.ster- 

f? Indicates that the Item Difficulty Exceeded the 1% Upper Confidence 
m Jt ^^^^ iyH'f ^® Asterisk Indicates that the Difficulty was 
Higher than the 5/^ C.L. but no Higher than the 1% C.L. Two Aster- 
isks Indicate that the Difficulty did not Exceed the 5^ C.L. 
(Decimal Points have been Deleted.) 



Grade Grade Grade Grade 

5 6 Item 5 6 



^ u % u 1^. i 



5 41 40 27 42 

7 ^4 28 76 i? 

8 i< l\ 29 55 66 
o <^ ?2 30 23*^' 30 

63 67 31 20^-'* i8** 

1? 1^ 32 48 47 

12 18*-' 30* 54 fz,.:-* 08** 

i2 ?r §ft 21 23- 28** 

Zf ?S 36 21*-* 23** 

J-5 63 60 37 20*** 25** 

16 78 80 38 39 

17 11-* 12** 39 ^2 -J? 

18 p.^* 19** 46 33 21 

19 47 59 41 22*- 22** 

20 35 45 42 24*-^^ 21** 

21 68 72 4p 24** 26** 

22 51 55 44 51 60 

45 24*''* 24*" 



Table a6 

Test 6 - Item Difficulties Under the NP-Condltions, Per Grade. No Aster- 
isk Indicates that the Item Difficulty Exceeded the 1% Upper Confidence 
Limit Around (l/k). One Asterisk Indicates that the Difficulty was 
Higher than the 5,'? C.L. but not Higher than the l)i C.L. Two Aster- 
isks Indicate that the Difficulty did not Exceed the 5% C.L. 
(Decimal Points have been Deleted.) 



Item 



Grade 
4 



Grade 
5 



Grade 
6 



Item 



Grade 
4 



Grade 
5 



Graivs 
6 



1 


07** 


09** 


05** 


2 


27** 


31 


34 


2 


79 


81 


86 




88 


89 


91 


1 


02** 


03** 


03** 




80 


81 


87 


7 


64 


65 


72 


8 


50 


47 


47 


9 


79 


81 


82 


10 


?^ 


50 


46 


11 


41 


43 


44 


12 


55 


65 


70 


11 


62 


64 


70 




38 


30* 


27** 


il 


26** 


36 


39 




49 


52 


60 


17 


52 


52 


53 


18 


37 


44 


51 


19 


37 


46 


45 


20 


18** 


19** 


18** 


21 


78 


86 


91 



22 

u 

25 
26 

27 
28 

29 
30 

31 
32 

33 
34 

36 
37 
38 

9 
0 
41 
42 



64 


76 


78 


41 


42 


43 


45 


50 


65 


24** 


25** 


24** 


54 


56 


61 


50 


50 


61 


23** 


33 


48 


19** 


21** 


26** 


62 


73 


82 


16** 


22** 


23** 


38 


52 


58 


36 


32 


37 


16** 


19** 




49 


58 




41 


40 




13** 
45 




iy** 


43 


42 


29* 


. 36 


40 


35 




46 


43 


^9 


46 


31 


33 


33 



APEENDK B 



For Each Test, Difficulty Coefficients (Tables E1-B6) Under P and UP 
Conditions (dp, d^^^) , Passage Dependency Index 2 (PDI = 1-d^ ) 
and Passage Dependency Efficiency Index ^ 
(E2 = l-V^p). 



Table Bl 



Test 1 - Difficulty Coefficients Under P and NP-Conditions (dp, d™). Passage 
Dependency Index 2* and Passage Dependency Efficiency Index tE2)** 
(Coxobined Across Gzadcs; Decimal Points are Deleted*) 



Item 


^2 






PDI, 




Item 


E2 




iMir 


irDI, 

** 


1 


14 


95 


82 


18 




39 


47 


76 


40 


60 


2 


66 


89 


31 


70 




40 


66 


55 


19 


82 


3 


45 


92 


51 


49 




41 


27 


43 


31 


69 


4 


10 


94 


35 


15 




42 


46 


58 


31 


69 


5 


51 


91 


45 


55 




43 


30 


38 


26 


74 


6 


10 


87 


79 


21 




44 


72 


54 


15 


85 


7 


53 


90 


42 


58 




45 


68 


45 


15 


86 


8 


21 


89 


70 


30 




46 


32 


52 


36 


65 


9 


42 


81 


47 


53 




47 


53 


55 


26 


74 


10 


80 


83 


17 


83 




48 


49 


68 


35 


65 


U 


25 


85 


64 


36 




49 


29 


61 


44 


57 


12 


27 


88 


64 


36 




50 


62 


57 


22 


78 


13 


19 


83 


67 


33 




51 


23 


66 


51 


49 


14 


10 


90 


81 


19 . 




52 


29 


62 


44 


56 


15 


38 


90 


56 


44 




53 


32 


66 


45 


55 


16 


28 


88 


64 


3o 




54 


49 


49 


25 


75 


17 


32 


87 


60 


41 




55 


37 


54 


34 


66 


18 


90 


78 


08 


92 




56 


-08 


61 


66 


34 


19 


38 


69 


43 


57 




57 


-07 


34 


64 


36 


20 


-3.57 


08 


32 


68 




58 


45 


34 


18 


82 


21 


40 


88 


53 


47 




59 


30 


38 


26 


74 


22 


60 


87 


35 


65 




60 


34 


40 


26 


74 


23 


91 


70 


06 


94 




61 


39 


33 


20 


80 


24 


20 


68 


55 


45 




62 


-02 


48 


49 


:>i 


25 


83 


75 


13 


87 




63 


-04 


39 


41 


59 


26 


58 


64 


27 


74 




64 


36 


37 


24 


76 


27 


61 


76 


30 


71 




65 


53 


37 


17 


83 


28 


31 


76 


53 


47 




66 


18 


15 


18 


82 


29 


72 


73 


21 


80 




67 


03 


24 


23 


77 


30 


31 


61 


42 


58 




68 


02 


38 


38 


62 


31 


34 


85 


56 


44 




69 


19 


41 


49 


51 


32 


41 


76 


45 


55 




70 


-07 


12 


13 


87 


33 


31 


82 


57 


43 




71 


58 


40 


17 


83 


34 


30 


56 


40 


60 




72 


48 


32 


47 


53 


35 


30 


71 


49 


51 




73 


63 


18 


29 


71 


36 


62 


59 


23 


77 




74 


13 


25 


22 


78 


37 


51 


60 


29 


71 




75 


-1.20 


13 


29 


71 


38 


38 


75 


47 


53 















** ^2 - ^-^^^e 



U8 



Table B2 



Test 2 - Difficulty Coefficients Under P and UP Conditions (d^, d. ) 

Passage Dependency Index 2* and Passage Dependency ^ ^ 



Efficiency Index (E^ 



Item 


^2 








^P 


1 


86 


<^ 


2 


90 


93 


3 


hS 


74 


k 


61 


81 


5 


66 


62 


6 


Ik 


71 


7 


59 


82 


8 


i!8 


80 


9 


81 


85 


10 


70 


83 


11 




69 


12 


1(2 


67 


13 


18 


h3 


Ih 


56 


bk 


15 


27 


60 


16 


ho 


73 


17 


38 


79 


16 


3^ 


71 


19 


21 


59 


20 


30 


hh 


21 


0^ 


38 



dfjp 



14 
10 
38 
31 
21 

19 

34 

42 

16 

25 

38 

39 

35 

37 

43 

44 

49 

47 

46 

31 

39 



PDI. 



87 
90 
62 

67 
79 
82 
66 
58 
84 

75 
62 
61 
65 
63 
57 
56 
51 
53 
S^4 

69 
61 



Item 



22 

23 
24 

25 
26 

27 
28 

29 
30 
31 
32 
33 
34 
35 
36 
37 
38 

39 
40 
41 
42 



03 
61 
17 
16 
77 
08 
45 
08 
11 
28 

45 
10 

08 
61 

69 
64 
76 
68 
58 
27 
29 
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EDL 


26 


26 


75 


18 


29 


71 


84 


69 


31 


76 


64 


36 


85 


19 


81 


78 


72 


28 


71 


39 


61 


58 


53 


47 


49 


54 


46 


37 


26 


74 


54 


30 


70 


43 


38 


62 


53 


49 


5:. 


75 


29 


7j. 


76 


23 


77 


64 


23 


77 


65 


15 


85 


55 


17 


83 


52 


22 


78 


30 


22 


78 


29 


21 


80 
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Table B3 

^ ■ Sfsa^f DLenr'^T^' ^ ^'^^ ^ Conditions (d , 

Passage Dependency Index 2* and Passage Det^endency P 

Efficiency Index (Eg)*^- ~ 




i'DI. 



1 


35 


2 


11 


3 


'29 


k 


25 


5 


8U 


6 


32 


7 


28 


8 


if5 


9 


l»8 


10 


li8 


U 


6k 


12 


36 


13 


57 


11^ 


15 


15 


56 


16 


25 


17 


01 


18 


k5 


19 


h5 


20 


67 


?1 


ks 


?.2 


31 


23 


14 


2k 


19 


25 


62 


26 


k7 


27 


21 


28 


70 


29 


27 


30 


58 


it 


Fijig = 1- 




= 1- 



95 
92 
84 

85 
88 
75 
76 

79 

kk 

92 

6k 

63 

6k 

88 

68 

66 

60 

58 

53 

82 

81 

72 

84 

83 

57 

54 

76 

58 

72 

59 



62 
82 

59 
64 
14 
51 
54 
44 

23 
48 

23 
41 

27 
75 
30 
50 

59 

32 

29 

27 

46 

50 

72 

67 

21 

28 

60 

17 

52 

25 



39 
18 
41 
36 
86 

49 

46 

56 

77 

52 

77 

59 

73 

25 

70 

50 

41 

68 

71 

73 

54 

50 

28 

33 

79 

72 

40 

83 
48 

75 



31 
32 
33 
34 
35 
36 
37 
38 

39 

40 
41 
42 
43 
44 
45 
46 
47 
48 

49 

50 

51 

52 

53 

54 

55 

56 

57 

58 

59 
60 



73 

49 

h3 

26 

73 

56 

42 

58 

18 

38 

71 

11 

49 

39 

49 

09 

h9 

45 

62 

43 
23 
55 
28 
48 
42 
28 

33 
14 

17 
17 



61 
57 
54 
78 

69 

71 

42 

60 

40 

57 

76 

^3 

59 

46 

43 

64 

54 

38 

66 

52 

57 

49 

52 

46 

46 

43 

35 

29 

34 

27 



16 


84 


29 


71 


30 


70 


58 


42 


19 


81 


31 


69 


24 


76 


25 


75 


33 


67 


35 


65 


22 


78 


38 


62 


30 


70 


28 


72 


22 


78 


58 


42 


28 


72 


21 


79 


25 


75 


30 


70 


43 


57 


22 


78 


37 


62 


24 


76 


26 


"h 


31 


69 


24 




25 


7'j 


28 


72 


22 


78 
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Table Bk 

Test k - Difficulty Coefficients Under P and KP Conditions (dp, d ) , 
Passage Dependency Index 2^ and Passage Dependency ^P 
Efficiency Index (E )** 



HP 



PDI, 



Item 



E. 



Item E, 



1 


47 


98 


2 


49 


76 


3 


15 


92 


k 


Ik 


94 


5 


ko 


80 


6 


33 


92 


7 


37 


75 


8 


08 


88 


9 


80 


84 


10 


08 


68 


n 


38 


40 


12 


16 


84 


13 


20 


46 


ih 


37 


84 


15 


06 


47 


16- 


Ik 


83 


17 


on 


53 


18 


13 


76 


19 


70 


71 


20 


18 


47 


21 


kl 


77 


22 


35 


60 



* ^^^2 = l-'\lP 

4w E = 1-d /d 
2 UP P 



51 


49 


23 


39 


61 


24 


79 


22 


25 


81 


19 


26 


48 


52 


27 


61 


39 


28 


47 


53 


29 


81 


19 


30 


17 


83 


31 


74 


26 


32 


25 


75 


33 


71 


29 


34 


37 


63 


35 


53 


47 


36 


50 


51 


37 


72 


29 


38 


49 


51 


39 


66 


34 


40 


21 


79 


41 


38 


62 


42 


41 


59 


43 


39 


61 


kk 






45 



70 


60 


18 


82 


08 


78 


72 


23 


12 


80 


70 


30 


36 


72 


46 


r'-i- 


12 


52 


^46 




17 


55 


46 


54 


72 


55 


16 


84 


17 


69 


57 


4J 


38 


39 


25 


76 


12 


74 


83 


17 


14 


56 


63 


37 


35 


64 


41 


59 


77 


60 


14 


86 


14 


44 


50 


50 


36 


83 


45 


55 


03 


66 


68 


32 


71 


62 


18 


82 


07 


56 


60 


41 


12 


49 


43 


57 


06 


61 


57 


43 


12 


67 


75 


25 


21 


48 


38 


62 


06 


40 


38 


'2 
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Table B5 

Test 5 - Difficulty Coefficients Under P and UP Conditions (d d ) 
Passage Dependency Index 2* and Passage Dependency^ p' 
Efficiency Index (E )** 



xcem 



Eg 



1 


07 


2 


14 


3 


k6 


h 


df 


5 




6 


52 


7 


21 


8 


16 


9 


07 


10 


22 


U 


25 


12 


k& 


13 


56 




09 


15 


28 


16 


08 


17 


81 


18 


6k 


19 


38 


20 


1(6 


21 


13 


22 


Ih 


23 


50 



86 


80 


88 


76 


55 


81 


h9 


51 


71 


kl 


84 


hi 


88 


69 


83 


69 


70 


65 


79 


61 


79 


60 


ko 


2k 


65 


29 


90 


83 


85 


62 


86 


79 


59 


11 


53 


19 


85 


53 


7h 


ko 


81 


70 


61 


53 


76 


38 
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20 
2k 

19 
k9 
60 
59 
31 
32 
35 
39 
ko 

77 
71 
18 
38 
21 

89 
81 

k7 
60 
30 
kl 
63 
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2k 

25 

26 

27 

28 

29 

30 

31 

32 

33 

3h 

35 
36 
37 
38 

39 

ko 
hi 

k& 
kS 
1(4 
45 



23 
28 

51 
02 
08 
12 
41 
56 
19 
49 
78 

13 
44 
68 
35 
49 
10 
47 

m 

20 
22 
60 



69 
45 
65 
51 
85 
69 
45 
44 

59 
69 
49 
30 
40 

69 
64 
67 
41 
41 
44 
21 
71 
59 



53 
33 
32 
50 
78 
60 
26 

19 
48 

35 
11 
26 
22 
22 
42 
34 

37 
22 

23 
25 
55 
24 



47 

6e 

69 

50 

22 

4o 

74 

81 

52 

65 

89 

74 

78 

78 

58 

66 

63 
78 

77 
76 



'^2 = 
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Table b6 

Test 6 - Difficulty Coefficients Under P nnd KP Conditions (d , d ), 
Passage Dependency Index 2* and Passage Dependency ^ KP 
Efficiency Index (E 

2 



Item 


^2 


d 

r 


d 

rJP 


PDI5 


Item 


2 






2 


1 


92 


89 


07 


93 


22 


09 


80 


73 


27 


2 


65 


89 


31 


69 


23 




78 


k2 


r3 


3 


06 


78 


82 


18 


2k 


02 


3k 


53 


h7 




Oi^ 


86 


90 


10 


25 


68 


77 


2k 


76 


5 
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65 


03 


98 


26 


27 


78 


57 


h3 


5 


07 


88 


83 


17 


27 


11 


61 


3k 
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7 


11 


75 


67 


33 


28 


^3 


61 


35 


66 


8 


Uh 


86 


kS 
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29 


39 


36 


22 


73 


9 
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89 


81 


19 


30 


11 


81 


73 


2Q 


10 


08 


51 


k7 


53 


31 


ko 


3k 


21 


'.'9 


11 


14 


50 


ks 


57 


32 


20 


62 


50 


50 


12 


09 


70 


63 


37 


33 


52 


73 


35 


6^ 


13 


10 


73 


65 


35 


3^ 


67 


55 


18 


82 
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19 


39 


32 


69 


35 


1(2 


ko 


56 


kk 


15 


56 


78 


3k 


66 


36 


17 


k6 


38 


62 


16 


15 


63 


3k 




37 


3k 


32 


15 


86 


17 


2k 


69 


52 


kQ 


38 


22 


55 


h3 


57 


18 


U2 


75 


kh 


56 


39 


05 


37 


33 


65 


19 


k6 


79 


ks 


57 


ko 


16 


1^7 


39 


61 


20 


6k 


50 


18 


82 
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16 


55 


k6 


5h 


21 


07 
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35 


15 
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32 


32 
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Appendix C 
Statistics for Passage Dependency 
of Test Items 

Introduction 

Tests of reading conxprehension purport to measure how well a 
student understands what he is reading. Many of 'these tiests employ 
questions to ascertain the degree of this understanding. At ftice 
value, these tests are very similar to any achievement test using 
the familiar multiple-choice format. In the case of r^addng contpre- 
hension tests, ho\7ever, the tacit assumption exists that there is a 
direct relationship bet\7een the reading of the passage or the story 
and the ability to answer questions about it. In the case of a great 
many reading test items from standardized tests* this is a faulty 
assuPiption. It has been well demonstrated that the probability of 
a correct answer prior to reading the paragraph exceeds chance in 
the case of most reading comprehension questions. (Preston, 1964; 
Farr and Smith, 1970; Bickley, Weaver and Ford, 1968; Liitchell, 
I9S7; Tuinman, 1970 ; Weaver and Bickley, I967). 

It must be pointed out that items with a relatively high pas- 
sage independency (i.e., ans\7erability with no passage being read) 
are not necessarily invalid. A student faced with answering such 
an item may actually use the information in the passage (which would 
be available under normal test condition.^ )^ even if he could have 
ansi/ered it by relying on extrinsic info3nnation) such as general knowlr 



syntactic cues, infoxiaation present due to particular item sequences 
and the like. 

The extent to which students will skip passages in actual test- 
ing situations is largely an unlmovm factor. Indirect evidence sup- 
porting the assumi^tion that students can be teiapted not to utilize 
information present in the passage is provided by Tuinman, 1972a and 
Tulnman (1972b). 

The presence of passage independency in a reading comprehension 
test thus creates uncertainty about the validity of any measurement 
taken with this test. The problem is complicated by the fact that it 
is not at all clear to what extent the ability to answer .questions 
without having read a passage is related to the ability to ansvrer 
questions after reading a passage. (Preston, 1961^; Tuinman, 1970 and 
Eurich, 1931). 

The thesis of .the present paper is that a "good" reading com- 
prehension item need not only meet such generally accepted criteria 
as adequate item difficulty and item reliability, but tliat such items, 
in addition, must be tested against criteria derived from the neces- 
sity to maximize passage dependency in order to reduce unceitainty about 
the content validity of a specific measurement. Currently, test de- 
velqpers tend not to apply such criteria and little discussion of 
them is available in the literature* The remainder of this paper 
is devoted 1) to a description of a procedure to estimate the degree 
of 'Validiiy-uncertainty" of a test (or an item); 2) to illustrations 
using fictitious and actual item data and 3) to a description of the 
problems and assucptions associated with this procedure. 
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Scoe basic assuciptions and formulas 

It might be argued that all one needs to ]mow about the passage 
dependency of a reading comprehension item is the probability of ansr/er- 
ing it correctly vrhen no passage is presented* If this probability does 
not exceed chance (usually l/k, where k = number of optlcns in a multi- 
ple-choice item), so goes the argument, the item is "good" in respect 
to passage dependency/ This approach is too simplistic, hoi/ever, as 
is quickly demonstrated by the item which has a difficulty of l/k in 
both the passage present and the passage absent conditions. The fol- 
lotfing discussion will point out further complications. First* a number 
of statistics should be defined: 

Pg^ = the proportion of correct responses to item i 
under the no-passage (UP) condition 

Pb ^ Pa the proportion of incorrect responses to 
item i under the UP condition 

Pc = the proportion of correct responses to item i 
under the passage (p) condition 

Pd = - Pc> "the proportion of incorrect responses to 
item I under the P condition 

To obtain and Pj^, the items are administered to a sample from 
a given population of test takers who ansvxer the questions without 
being able to read the passages on which the questions are based. 
The statistics p^ and pa are obtained fi:om an independent sample 
of subjects from the same population. Marks and Noll (I967) obtained 
estimates of the contribution of passages to items based on them by 
administering the items t\rice to the same group of respondents. The 
procedure proposed here avoids both the measurement and practical 
problems associated Tdth this approach. 
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Ss who ans\7er the items under the HP condition must either act 
on the basis of scnne extrinsic information (i*e«, information not 
derived from the passage) or they are guessing. Accordingly, V7e may 
vrrite : 

Pa = Pal +Pa2 ^here , 1) 

Pal = the proportion of correct responses to item i 
oaset on guessing 

Pa2 = the proportion of correct responses based on 
e:ctrinsic infoimtion. 

Due to the peculiarities of the IlP-condition some behaviors vhich .rc 

ncroolly considered to be part. of tho i\«uoc;aii:i" bekavicr (Tinkelman, 1971) 

are currently included under the category of responding on the basir. 

of extrinsic information-- for e:canrple, maldng use of semantic or 

sinitactic cues available frcM the wording of the question. It is 

necessary to assume that there is merit in the use of conventional 

correction formulas for guessing. The likelihood of this assumption 

being correct is supported by the fact that many essentially non-guessing 

behaviors are e:ccluded from the guessing component under the definitions 

employed here. Later in the paper this issue is discussed further; for 

the moment it is assumed that p^ can be estimated from the proportion 

of wrong ans\Ters (Pb or 1 - p^), using the logic of a correction 

for guessing formtila. Thus, 

Pal = Pb/(k-l) = (l-Pa)/(k-l) 2) 
V7here k = the number of options per item. 
It then follows that p^^ or the proportion correct responses 
based on extrinsic information is given by 3) 
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Pa2 = Pa - Pal 3) 
Expressed directly in terms of p^^, Pg^2 is given by k) 

Pa2 = l^(Pa) • 1 /(^-^) 
Analogous to the partitioning follcfwed above, the statistics obtained 

ftom the administration of the test items under the P-condition 

result in: 

p^ = the proportion of correct responses 

Pel** the proportion of correct responses based on guessing 

Pc2= the prq?ortion of cox'rect responses based on information 
extrinsic to the passage 

Pc3" proportion of correct responses based on the passage. 

Thus, 

Pc = Pel Pc2 + Pc3 5) 
Again, p^ can be calculated frota the proportion of incorrect responses, 

Pd = 1 - Pc- Thus, 

Pel = (1-Pc)/(1^-1) 
Since the passage usually does provide at least some information not 

contained in the questions only, Pai ^ Pcl> except under the rare 

condition that p^ = p^. 

It is impossible to partition Vc2 Pc? ^i^en only proportional 

data Pa, Pb> Pc> Pd- means that no direct estimate of th3 

contribution of the passage to the responses to an item or set of items 

is available. Hoi^ever, it is possible to calcxilate in a straight- 

fonmrd manner a number of statistics which may be of use in deciding 

on the quality of the item or group of items under consideration. 



These statistics are: 

a) majdinum contribution of e:rtrinsic inforriiation 
under the P-condition 

b) Pfljijj, the minimum contribution of the passage 

c) Pjuax^ '^^^ maximum contribution of the passage 

If none of the students responding to an item is able to utilize any 
of the information in the passage, extrinsic information T7ill be 
e2:ercising maximum influence • This maximum influence is indexed by 
Pa2* Thus, 

Emax = Pa2 7) 
The minimum contribution of the passage is given by 8) 

Pmin = Pc - Pa - Pa2 8) 
Equation 8) foUovxs from 5) given the identity Pg^2 = Pc2» which holds 
in the case of minimum contribution of the passage • 
^min ^® directly calculated froa Pg^ and p^ using 9) 

Pmin = ^(Pc - Pa)/(k-l) 9) 
The maximum contriburbion of the passage occurs only when the students 
do not utilize any extrinsic information when answering the items, 
hence when p^ = 0. In this case the value of p^3 equals 

Pmax = Pc - Pcl.^ or 10) 

IWx = ^(Pc) - ^ /(k-1) 11) 
It may be redundant to observe that 

" ^max " ^lain ^-2) 
Since the statistics calculated above are all linear transformations 

of the basic probabilities p^^ and p^, upper and lov/er confidence 



limits for these probabilities can be substituted in the 
in order to arrive at confidence regions for the various 
interest. Furthermore, a correction for the calculation 
must be aj^lied in the case of omitted responses • 

An illustration with fictitious data. 



Insert Table C7 about here 



In Table C7> hypothetical values of p and p are paired and the values 

a c 

of the corresponding E » and P are presented. First, from 

max max 

equation it is clear that E depends solely on the value of p • In 

max a 

a senses therefore, this statistic does not contribute any new knowledge 
about an item. E^^ i£ useful as an indication of the maximum proportion 
of correct responses to an item und^er the P-condition which could be at- 
tributed to extrinsic knowledge • If, for instance, under the KP-condition 
kOfo of the Ss ansvrer item 1 correctly, this does not mean that imder the 
P-condition kO^ of the correct responses could be due to extrinsic inf 
mation. Rather, this proportion cannot be higher than .20 as can be s. r 

in Table C7. The P statistic fulfills a similar function. It is ex- 

max 

clusively dependent on and therefore adds no information in terms of 

relative relations among the Items. Yet, P i£ useful as an index of 

^ max 

the absolute proportion of correct responses under influence of the pas- 
sage. 

lb was noted above that a determination of an item's quality in terms 
of passage dependency can not be made only on the basis of p . Given a 
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formulas above 
statistics of 
of p and p 
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multiple-choice test, a number of items may have a p value of .25. The 

a 

p3 statistic is sioiply of no help in discriminating among these items 

in terms of the extent to which the passage controls the responses. lii 

this particular case. eith» P or P provides the information neces- 

min max 

sary to make a decision about an item's desirability. For norm-referenced 

measurement an item with p^ = .25 and p^ = .60 might be a desirable item, 

a c 

whereas for criterion -referenced measurement the combination p = . 25 and 

a 

p =1.00 might be preferable, 
c 

Table C!7 shows negative values of P .It seems illogical to talk 

min 

about negative minimal contributions of the passage in answering questions 

based on that passage. Yet, these negative P • 's are realistic and in- 

min 

dex a situation which is not uncommot . Some reading coinprehension items 

are easier with no passage present than with the passage present. This 

is true, of course, for miskeyed items, but is also true when a passage 

contains ambiguous information or when it leads some students not to 

choose the obviously right answer. This case is illustrated by the sev. 

ond set of p^ and p^ values in liable C7. 

A p ^ equal to -.20 indicates that at the most 20^ of the incorrect 
mm 

responses would be a function of misleading information in the passage. 

This interpretation of negative P »s is illustrated best by the item 

min 

with p^ = 1.00 and p^ = .25. Here 25?& of the responses Tjould be correct 

presumably by guessing and all of the remaining responses are wrong as a 

function of ambiguous or misleading information in the passage. 

It may be further observed that P can only equal 1.00 if p = 

.25 and p =1.00. In this case .the minimum and maximum contribution of 
c ' 

the passage are identical. However, P can be 1.00 while P ^ =0. 

max min 
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This is the case v?hen both p and p are equal to l.OO. These observa- 

a ^ 

tions indicate an iinportant point. It was argued earlier that a p 

a 

value higher than l/k (where k = number of options) does not necessarily 
mean that an item is invalid, i.e., that responding to it under the P- 
condition does not involve utilization of the information from the passage. 
TThat matters is that uncertainty about hm the item was answered, (i.e., 
whether passage information was used or not) should be minimal. This un- 
certainty is indexed directly by the difference between P and P , and 

min max 

therefore by E .The smaUer E , the less uncertainty about the ac- 
max max 

tual degree to which the passage influenced the responses. Yet, as the 

second set of five items in Table C7 indicate, miniiaal E *s are possible 

caax 

under a variety of conditions; zero uncertainty is possible both with 

0.0 and 1.00 minimal contribution of the passage. Relatively lev; E ' 

max 

are only meaningful in relation to their corresponding p ,mlues. 

min 

This particular point will be further illustrated in the next section of 
this paper. 

An illustration with real data 

The statistics described above were calculated for six different 
standardized tests of reading comprehension, all of which are frequently 
used in routine assessment of reading performance in the public schools. 
Four of the tests were administered to a sample of Ifth, 5th and 6th 
graders. One test v/as suited for the kth grade exclusively and one test 
was administered in only the 5th and 6tli grades. In the HP-condition 
each grade provided kCO subjects. In the P-condition 200 subjects per 
grade and per test were used. The total sample of subjects used \i^s 
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slightly over 9,000. A full description of the sample and the proce- 
dures follov7ed is provided in the inain body of this report. Table c8 
contains the validity statistics calculated for these tests. The entries 
are averages for the number of items indicated. In some cases this num- 
ber is smaller than those present in the test. This is a function of the 
fact that only the items completed by all the Ss in the P-condition 
included in the present calculations. 



Insert Table C8 about here 



A number of observations need to be made. Imagine that the decision 
called for is to select the test that gives the most guarantees that the 
responses to the items are a function of information in the passage, 
at the same time not violating other rationales for selecting tests. 
First, the test difficulties do not vary much with the exception of Test 
1. This test is the Kelson Reading test, rather a speed test, and since 
only the first hO items are considered here, the loi^ difficulty (.77) is 
not surprising. None of the tests fares very well on the question of pas- 
sage dependency of the items. The best in the group. Test 2 (California 
Achievement Battery Reading, Level 3A) has an average difficulty under 
the NP-condition of .3i^. This means that, at most, 1^ of the correct r. - 
sponses under the P-condition could -be due to extrinsic information. 
Test 3 (a subset of SBk Reading Test iteras) produces about as imch 
guaranteed responding under influence of the passage as does Test 2 (.37 
versus .38) but allows for the possibility that the passage may determine 
a higher percentage of correct responses (.59 to .52). The price for buy- 
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ing more passage control by selecting Test 3 as the first choice, hovever, 

is more uncertainty about the validity of the responses of a given set of 

respondents. Tiie comparison of Tests 1 and 6 illustrates another selection 

problem. Both he.ve identical E values. Ho\7ever, Test 1 seems prefer- 

max 

able in vie\7 of its higher value, provided that the low difficulty 

level of the test does not constitute an a priori reason to reject the 

test. This, it appears, would depend on other criteria set for test usage, 

criteria unrelated to passage control over the responses. 

The necessity to relate values to P^^^ values as illustrated 

above suggests the use of a P . /e ratio. 

min max 

Insert Table C9 about here 



Table C9 illustrates the use of the validity indices proposed here 
for the selection of items, as opposed to those for tests. Included al- 
so is the ratio of P to E . Briefly, Tfeible C9 suggests the follow- 

mm max 

ing observations: 

1. The difference betv/een p and p^ is not enough to determine the 

c a 

quality of an item in terms of passage dependency characteristics. Cri> 
pare Items 1 and Ifl, for example, on Test 1. 

2. Negative E values arise when relatively easy items contain a 

max / 

false option which seems a very good choice when there is no information 

from the passage present. This leads to artifically inflated P values. 

min 

The actual contribution of the passage in these cases is given by P 

max 

with zero uncertainty about the passage's contribution existing. (See 
Item 18, Test 1.) 
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3. Items of equal difficulty under the P-condition may differ 

vastly in difficulty under the KP-condition. (Conrpare Items 8 and 9, 

Test 2). These differences are most meaningfully reflected in the 

P . values of the items . 
min 

k» Nearly identical E values may obscure differences between 

max 

items reflected in the P„.„/E ratio (item 1, Test 1 and Item 8, 

niin' max 

Test 2). Given the acceptability cf a p value equal to .95, Item 1 is 

c 

to be preferred over Item 8 on grounds of the higher ratio* 

Table C7 indicated that for a given p -value. P . and P are 

a min max 

highly correlated, whereas the correlation beti/een E and these two 

max 

statistics is zero (in absence of variance among E values for a par- 

max 

ticular p -value). In a set of items with varying combinations of p 
^ a 

and p^ values, hoi^ever, the relations are less sinrple. Table CIO con- 
tains a correlation matrix based on the completed items for Test 5. 
This matrix is representative of those coniputed for the other tests. 



Insert Table 10 about here 



It must be borne in mind that since E is a linear transformation of 

max 

Pq> and P is such a transformation of p , correlations bet^-7een E 



max c 



max 

and any variable and between P and any variable also hold for that 

max 

variable and p and p .respectively. The most important conclusion to 
a c 

be drawn from Table CIO is that, for a particular set of items, each of 
the variables calculated provides different information relating to the 
issue of pr.GGaL»e dependency of a particular item. 
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Discu? ^sion - 

The mo'or point brought out by the preceding analysis is that it 
is insufficient to consider the passage dependency of reading coicpre- 
hension items solely in terms of their difficulty under the KP-condition. 
Wo attempt has been made to provide a stq)-by-step procedure for evalra- 
tion of i-^ems in this respect. Rather, a number of indices have been dis- 
cussed TuhiLch under certain conditions may be of use in comparing items or 
tests in terms of their measuring behaviors which are under control of 
the information in the passage. 

The fact that the statistics proposed are linear transformations ^ r 

p , p and p^-p^ indicates that for practical decisions the latter 
a c c a 

quantities might be used just as well P , P or E . For example, 

min max max 

in many instances it might suffice to look for items with p values with- 

c 

in an acceptable range on large values of p -p . The reason for consider- 

c a 

ing the three new statistics proposed here is, first of all, the fact that 
they emphasize an essential problem with reading comprehension items (i.e., 
the contribution of the passage to the probability of a correct response). 
Secondly, the statistics allow an expression of the limits of this con- 
tribution in terms of percentages which are considered to be more meaning- 
ful by this author than the mere proportions of correct responses under 
the P and NP-conditions. 

It already has been acknowledged that the adequacy of the analysis 
outlined above depends on the degree to which application of the correc- 
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matter 



tion-f or -guessing formula laay be assumed to be correct. This assumptica 

bears directly on the calculation of p and p . As Tinkelman (1971) 

al cl 

points out, the term "guessing" quite loosely refers to an array of be- 
haviors, many of which involve responding in tenas of partial information 
rather than essentially non-systematic response behaviors. To the degree 
that this is the case, the application of correction-for-guessing formula's 
is questionable. In terms of the present analysis this conclusion must be 
interpreted in the light of the following observations. 

First, the concept of "extrinsic infornation" includes all utiliza- 
tion of '"partial" information which normally would be thought of as part 
of "guessing." The residual behaviors included in "guessing" are more 
likely to be of a non-systematic nature than is the case in those applica- 
tions where no attempt is made to separate the various components of be- 
haviors leading to incorrect responses. 

Secondly, the formulas and concepts developed above may be used wi 
entire tests as vrell as with individual items. It stands to reason that 
the various estimates of responses based on a particular source of infor- 
mation are more reliable in the former case. This is true, too, in the 
case of the statistics yielded by the application of the correction-for- 
guessing formula. The assumption underlying this application in the case 
of a set of items is a weaker assumption. It is only necessary to assume 
that the sums of the responses for the v/rong options are equal across 
items and subjects. That is an assumption which is more likely to hold 
than its more stringent counterpart which must hold in the case of the 
application of the formula for correction with individual items: an 
equal number of lespondente selected each of the k-1 wrong options. 
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Thirdly, the analysis of reading cooiprehension items presently is in 
a rather primitive stage. If a more adequate analysis calls for distrac- 
tors that are basically equipotent, a very basic tenet of sound item writ- 
ing is merely reiterated. Currently, reading comprehension items too often 
contain distractors with very low potency. Such distractors not only low- 
er the general efficiency of these item^ h}xk, as shown, actively interfere 
with adequate assessment of these item' fi^ passage dependency characteristics 
since lack of equipotency of distractors reduces non-systematic select 
of options. 
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Test 1 - Values of Item Validity Staviistics Described in 
Appendix C for all Itent^. 
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Table C2 



Test 2 - Values of Item VaUdlty Statistics Described in 
Appendix C for all Items. 
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Table C3 

Test 3 - Values of Item Validity Statistics Described in 
Appendix C for all Items. 
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Table C4 

Test 4 - Values of Item Validity Statistics Described in 
Appendix C for all Items. 
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Table C5 

Test 5 - Values of Item Validity Statistics Described in 
Appendix C for all It^os. 
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25 


.27 


.17 


1.71 


.10 


26 


.53 


.45 


5.06 


.09 


27 


.34 


.01 


.04 


.3" 


28 


.80 


.09 


.13 


.'.i. 


29 


.58 


.11 


.23 


.47 


30 


.27 


.25 


13.36 


.02 


31 


.25 


.33 


-4.32 


-.08 


32 


.45 


.15 


.48 


.30 


33 


.58 


.45 


3.47 


.13 


34 


.33 


.51 


-2.73 


-.19 


35 


.06 


.05 


5.71 


.01 


36 


.19 


.23 


-5.87 


-.04 


37 


.59 


.62 


-17.96 


-.03 


38 


.52 


.29 


1.31 


.23 


39 


.56 


.43 


3.5C 


.12 


40 


.21 


.05 


.33 


.16 


41 


.21 


.25 


-5.94 


-.04 


42 


.25 


.28 


-8.79 


-.03 


43 


-.05 


-.06 


-14.00 


.004 


44 


.62 


.21 


.52 


.41 


45 


.45 


.47 


-25.29 


-.02 
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Table C6 

Test 6 - Values of Item Validity Statistics Tescribed in 
Appendix C for all Items. 



Item 



flsax 



^min 



^min/ 



E. 



lOBLX 



1 


.85 


1.05 


-U.5U 


.2k 


2 


.85 


.77 


9.97 


.08 


3 


.70 


-.06 


-.08 


.76 


k 


.81 


-.05 


-.06 


.86 


5 


.53 


.33 


-2.78 


-.30 


6 


.85 


.08 


.10 


.77 


7 


.67 


.11 


.19 


.56 


8 


.81 


.51 


1.66 


.31 


9 


.85 


.10 


.ll^ 


.7k 


10 


.35 


.05 


.18 


.29 


11 


.33 


.10 




.2K 


22 


.59 


.09 


.17 


.51 


13 


.63 


.10 


.18 


.53 


14 


.18 


.10 


1.11 


.09 


15 


.70 


.58 


4.91 


.12 


16 


.51 


.13 


.3k 


.38 


17 


.59 


.22 


.61 


.37 


18 


.67 


.kz 


1.65 


.25 


19 


.72 




2.05 


.23 


20 


.3h 




-4.76 


-.09 


21 


.88 


.08 


.10 


.30 



Item 



IQ&X 



^iain 



P • / 
^nax 



ma:c 



22 


.73 


.10 


.16 


.6'+ 


23 


.70 


.kl 


2.0U 


.23 


2k 


.39 


.01 


.Ok 


.37 


25 


.69 


.70 


-75.29 


-.01 


26 


.71 


.29 


.68 


.1*2 


27 


.k8 


.09 


.2k 


.39 


26 


.m 


.35 


2.76 


.13 


29 


.15 


.19 


'k.90 


-.0l^ 


30 


.75 


.12 


.18 


.6?. 


31 


.13 


.18 


-3.1'^ 


-.OS 


32 


.k9 


.16 


.k9 


.33 


33 


.6k 


.51 


3.88 


.13 


31^ 


.ko 


.k9 


-5.36 


-.09 


35 


.19 


-.22 


'.5k 


.kl 


36 


.28 


.M 


.61 


.If. 


37 


.09 


.23 


-1.62 


'.Ik 


38 


.kl 


.17 


.69 


.2k 


39 


.15 


.03 


.18 


.13 


ko 


.29 


.10 


.53 


.19 


kl 


.39 


.12 




o28 


k2 


.09 


-.01 


-.11 


.10 
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Table C7 

Illustrative Validity Statistics for Selected 

Values of p and p 
•^a -^c 





P 




p 


p 




c 


max 


min 


mx 


Pa= -25 


Pc= -25 


.00 


.00 


.1. * 




Pc= .to 


.00 


.20 


.20 




Pc= .60 


.00 




.1*7 




Pc= .80 


.00 


.73 


.73 




Pc=1.00 


.00 


1.00 


1.00 


Pa= •'^ 


Pc= .25 


.20 


-.20 


.00 




Pc= M 


.20 


.00 


.20 




Pc= .60 


.20 


.27 


.1^7 




Pc= .80 


.20 


.53 


.73 




Pc=i.oo 


.20 


.80 


1.00 




Pc= .25 


.33 


-.33 


.00 




Pc= .to 


.33 


-.13 


.20 




Pc^ .60 


.33 


.11* 






Pc= .80 


.33 


.to 


.73 




Pc=1.00 


.33 


.67 


1.00 


Pa=1.00 


Pc= .25 


1.00 


-1.00 


.00 




pc= .to 


1.00 


- .80 


.20 




Pc= .60 


1.00 


- .53 


.1*7 




Pe= .80 


1,00 


- .27 


.73 




Pc"1.00 


1.00 


.00 


1.00 



Table CG 

Mean Values of Validity Statistics for Six Tests 



Test 


Items 






P 

mln 




E 


1 


40 


.77 


.46 


.42 


.70 


.28 


2 


42 


.64 


.34 


.38 


.52 


.14 


3 


40 


.69 


.41 


.37 


.59 


.21 


4 


45 


.66 


.50 


.22 


.54 


.33 


5 


45 


.64 


.45 


.25 


.52 


.27 


6 


40 


.65 


46 


.26 


.54 


.28 
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Table C9 

Validity Statistics for Selected Items 



Item 
Nunber 


Pc 


Pa 


P . 
min 


P 

max 


^max 


9 /t? 

min max 


Test 1 














1 


.95 


.82 


.17 


.93 


.75 


.23 


5 


.91 


.45 


.61 


.88 


.26 


2.32 


18 


.77 


.07 


.93 


.70 


-.23 


-4.05 


Al 


,43 


.31 


.15 


.23 


.08 


1.84 


Test 4 














8 


.88 


.81 


.09 


.84 


.75 


.12 


9 


.84 


.17 


.90 


.79 


-.11 


- 8.43 


10 


.47 


.50 


-.04 


. .29 


.33 


- .12 


31 


.39 


.25 


.20 


.19 


-.01 


-29.60 



* This ratio was calculated with unrounded values of P and E 

UU.U max 
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Table CIO 

Correlation Matrix 
Test 5 (N«45) 



(1) (2) (3) 

mln ^inax ^max 



^max "'^^ — 

Pfflax -18 .71 — 

P /E -.10 .29 27 
oin max 



